Interlacing data in streaming industrial image data

ABSTRACT

A system may include a control system for controlling one or more operations of one or more industrial devices in an industrial system. The control system may receive streaming data comprising one or more visualizations representative of one or more live operational parameters associated with one or more industrial devices. The streaming data may include multiple image frames. The control system may also identify multiple datasets associated with the streaming data and generate multiple machine-readable images based on the multiple datasets. In addition, the control system may embed the multiple machine-readable images within the multiple image frames of the streaming data to generate updated streaming data and send the updated streaming data to a computing system that may extract the multiple machine-readable images from the updated streaming data.

BACKGROUND

The present disclosure generally relates to providing remote monitoringof equipment operations. More specifically, the present disclosure isrelated to systems and methods for providing multiple users across anorganization with remote accesses to live data updates related tovarious equipment or devices of a system.

Users (e.g., plant operators, maintenance personnel) may prefer to haveaccess to live data related to current operating conditions of variousequipment and devices of industrial systems. However, providing livedata access to multiple individuals or entities may involve increasedcybersecurity measures for reducing potential risks of the live datathat may be compromised by access via unauthorized personnel (e.g.,hackers). Moreover, sending raw live data to the users for remotelymonitoring a large amount of equipment and devices may consume asignificant portion of network bandwidth, resulting in various issuessuch as data latency, communication bottleneck, and increased data riskor network cost.

This section is intended to introduce the reader to various aspects ofart that may be related to various aspects of the present disclosure,which are described and/or claimed below. This discussion is believed tohelp provide the reader with background information to facilitate abetter understanding of the various aspects of the present disclosure.Accordingly, it is understood that these statements are to be read inthis light, and not as admissions of prior art.

BRIEF DESCRIPTION

A summary of certain embodiments disclosed herein is set forth below. Itshould be understood that these aspects are presented merely to providethe reader with a brief summary of these certain embodiments and thatthese aspects are not intended to limit the scope of this disclosure.Indeed, this disclosure may encompass a variety of aspects that may notbe set forth below.

In one embodiment, a system may include a control system configured tocontrol one or more operations of one or more industrial devices in anindustrial system. The control system may receive streaming datacomprising one or more visualizations representative of one or more liveoperational parameters associated with one or more industrial devices.The streaming data may include multiple image frames. The control systemmay also identify multiple datasets associated with the streaming dataand generate multiple machine-readable images based on the multipledatasets. In addition, the control system may embed the multiplemachine-readable images within the multiple image frames of thestreaming data to generate updated streaming data and send the updatedstreaming data to a computing system configured to extract the multiplemachine-readable images from the updated streaming data.

In another embodiment, a method using a control system may includereceiving streaming data comprising one or more visualizationsrepresentative of one or more live operational parameters associatedwith one or more industrial devices in an industrial system. Thestreaming data may include multiple image frames. The method may alsoinclude identifying multiple datasets associated with the streaming dataand generating multiple machine-readable images based on the multipledatasets. In addition, the method may include embedding the multiplemachine-readable images within the multiple image frames of thestreaming data to generate updated streaming data and sending theupdated streaming data to a computing system.

In yet another embodiment, non-transitory, computer-readable mediumstoring instructions that, when executed by one or more processors,cause the one or more processors to perform operations including:receiving streaming data that includes multiple image frames via anetwork; extracting multiple machine-readable images corresponding to aportion of the multiple image frames from the streaming data;determining one or more data values corresponding to one or more liveoperational parameters associated with one or more industrial devicesbased on the multiple machine-readable images extracted from thestreaming data; and storing the one or more data values in a database.

DRAWINGS

These and other features, aspects, and advantages of the presentdisclosure will become better understood when the following detaileddescription is read with reference to the accompanying drawings in whichlike characters represent like parts throughout the drawings, wherein:

FIG. 1 illustrates an example industrial automation system controlled byone or more industrial control systems and monitored by users via a datastreaming device, in accordance with an embodiment presented herein;

FIG. 2 illustrates example components of the data streaming device ofFIG. 1 , in accordance with an embodiment presented herein;

FIG. 3 illustrates a flow chart of an example method for providing livestreaming visualizations with access control using the data streamingdevice of FIG. 1 , in accordance with an embodiment presented herein;

FIG. 4 illustrates example live streaming visualizations based on themethod of FIG. 3 , in accordance with an embodiment presented herein;

FIG. 5 illustrates a flow chart of an example method for extractinghistorical data from the live streaming data, in accordance with anembodiment presented herein;

FIG. 6 illustrates a flow chart of an example method for identifyingdata trends based on extracted historical data using the example methodof FIG. 5 , in accordance with an embodiment presented herein;

FIG. 7 illustrates a flow chart of an example method for interlacingdata in the live streaming data, in accordance with an embodimentpresented herein;

FIG. 8 illustrates an example of a number of image frames withinterlaced data based on the example method of FIG. 7 , in accordancewith an embodiment presented herein; and

FIG. 9 illustrates a flow chart of an example method for processing thelive streaming data with interlaced data, in accordance with anembodiment presented herein.

DETAILED DESCRIPTION

One or more specific embodiments of the present disclosure will bedescribed below. In an effort to provide a concise description of theseembodiments, all features of an actual implementation may not bedescribed in the specification. It should be appreciated that in thedevelopment of any such actual implementation, as in any engineering ordesign project, numerous implementation-specific decisions must be madeto achieve the developers' specific goals, such as compliance withsystem-related and business-related constraints, which may vary from oneimplementation to another. Moreover, it should be appreciated that sucha development effort might be complex and time consuming, but wouldnevertheless be a routine undertaking of design, fabrication, andmanufacture for those of ordinary skill having the benefit of thisdisclosure.

When introducing elements of various embodiments of the presentdisclosure, the articles “a,” “an,” “the,” and “said” are intended tomean that there are one or more of the elements. The terms “comprising,”“including,” and “having” are intended to be inclusive and mean thatthere may be additional elements other than the listed elements.

Organizations (e.g., plants) may prefer to provide access to live datarelated to current operating conditions of various equipment and devicesof industrial systems for multiple individuals or entities (e.g., plantoperators, maintenance personnel) across the organizations. Such livedata access may enable the individuals to monitor operations of theequipment and devices in real time and to provide remote control orenable remote personnel to provide advice and help trouble shootingoperational issues based on the accessed live data. However, providinglive data access to the multiple individuals or entities may increasepotential risks (e.g., data breach) of the live data being compromised(e.g., accessed) by unauthorized personnel (e.g., third partycontractors, hackers). Furthermore, providing remote monitoring of alarge amount of equipment and devices of an industrial system bytransferring raw live data may involve a significant portion of networkbandwidth, resulting in data latency, communication bottleneck,increased network cost, and the like.

Embodiments of the present disclosure are generally directed towardssystems and methods for providing multiple users (e.g., individuals orentities) across an organization with remote accesses (e.g., viewingaccess) to live data updates related to various industrial equipment anddevices of an industrial system. For example, an industrial automationsystem may include certain control systems of the industrial system. Thecontrol systems may output image data representative of live datasetsassociated with operations of the industrial equipment and devices viasuitable image data ports, such as High-Definition Multimedia Interface(HDMI) ports. The image data may be provided to a live-streaming deviceor component, such that the multiple users may view the image data via alive-streaming broadcast. By presenting the live image data, the dataavailable to a recipient may be unstructured (e.g., in a manner thatdoes not include substantive operation-related values of respectiveindustrial equipment and devices). In this way, data security may beenhanced by using pure image data that excludes digital or substantivevalues (e.g., temperature, pressure) of operation-related values, suchthat even if the image data is compromised (e.g., hacked), it isdifficult to interpret the values presented in the image data ordetermine the relationship of the values to the equipment presented inthe image data. In addition, the live datasets may be viewable by themultiple users but may not be modified or altered by the multiple users.In this way, the industrial equipment and devices may be accessible tothe multiple users for view but may not be accessible to unauthorizedusers for modifications (e.g., editing or writing operation).

With this in mind, in some embodiments, one or more industrial controlsystems may be communicatively coupled to various industrial equipmentand devices of an industrial system (e.g., production line) implementedin a plant (e.g., manufacturing plant). The industrial control systemsmay monitor and collect live data (e.g., image data, audio data, logdata) representative of certain measurement values (e.g., positions,orientations, speeds, temperatures, noise levels, pressures, voltages,currents), operation values (e.g., parameters, settings), and operationstatuses (e.g., service logs, status reports, incident reports,warnings, or alerts) associated with operations of the variousindustrial equipment and devices. The industrial control systems maytransmit the collected live data to a data streaming device that mayprovide live streaming visualizations of the operations of the variousindustrial equipment and devices. The live streaming visualizations mayenable users (e.g., plant operators, maintenance personnel) to viewcertain live datasets via the live streaming visualizations. Forexample, a user device (e.g., computing device) may send a request tothe data streaming device to view certain live stream visualizationsthat present plant floor datasets that may correspond to respectiveoperations of certain industrial equipment and devices. The request mayinclude credentials (e.g., user credentials), and based on the receivedcredentials, the data streaming device may determine view access rightsassociated with the user. The data streaming device may then send a listof equipment and/or devices to the user device, such that the listcorresponds to a set of devices that the user's credentials permitaccess. Based on received user selections (e.g., selecting a device,selecting a visualization associated with the selected device), the datastreaming device may route streaming image data associated with theselected visualization to the user device for further applications(e.g., data analysis, data processing). Additional details with regardto providing live streaming visualizations of industrial equipmentoperations using the data streaming device will be discussed below withreference to FIGS. 3 and 4 .

In some embodiments, after receiving streaming data from a datastreaming device, the user device or other suitable computing system mayextract historical data from the streaming data based on identified datafields (e.g., temperature, pressure) over a time period. For example,the streaming data may include image data consisting of image frames andaudio data consisting of audio packets. The computing system mayassociate audio packets with image frames and extract portions of theimage frames based on certain identified data fields (e.g., temperature,or pressure) that may vary over a time period. Such identification mayutilize optical character recognition (OCR) technology to extract datafrom the image data (e.g., streaming video). The computing system mayacquire data field values (e.g., temperature or pressure reading) fromthe extracted portions of the image frames. Furthermore, the computingsystem may identify audio packets corresponding to the image framesassociated with the extracted portions. The acquired data field valuesand the identified audio packets may be stored as time series data in adatabase for further applications (e.g., data analysis, faultidentification, alerting, or recommendation). In this way, unstructureddata, such as the image data presented via the live streamvisualizations may be captured and stored in a structured format forfurther data processing and analysis. For instance, the computing systemmay receive time series data from the database and identify certain datatrends associated with respective equipment or devices based on receivedtime series data. The computing system may also predict certainoperational parameters of the respective equipment or devices based onidentified trends. If a predicted operational parameter (e.g. predictedpressure) of an equipment (e.g., a tank) is outside an expected range(e.g., predetermined range based on historical operational valuesmeasured by pressure sensors), the computing system may sendcorresponding commands to the equipment to adjust operations (e.g., toprevent over pressure). Additional details with regard to extractinghistorical data and identifying data trends from the streaming imagedata will be discussed below with reference to FIGS. 5 and 6 .

In certain embodiments, a control system (e.g., industrial controlsystem) may receive datasets (e.g., substantive data, measurement, anddigital values) from operation equipment or devices of an industrialsystem. Based on the received datasets, the control system may generatecertain machine-readable images (e.g., a bar code) and interlace themachine-readable images with streaming image data provided to users forview via a data streaming device. Such interlaced machine-readableimages may not be perceivable by the user, whereas a viewing device(e.g., computer, tablet) displaying the streaming image data may becapable of detecting the interlaced machine-readable images (e.g., bysampling the streaming image data at a specific rate). In this way,additional information may be embedded within image frames of thestreaming image data and provided to the user. In some cases, interlaceddata may be used to prevent data loss caused by network issues. Forexample, certain network conditions, such as a transmitting devicelosing network connectivity, may cause missing image frames from thestreaming image data. Using interlacing data technique described above,a buffer of image datasets (e.g., a gather of high-density imagedatasets consisting of machine-readable images) may be interlaced withinthe streaming image data, such that any missing image frames may beaccounted for and retrieved from one or more neighboring image frames.In this way, the control system may utilize the machine-readable imagesinterlaced with the streaming image data to ensure that the streamingimage data may provide expected image datasets to the user duringoccurrences of various network issues. Additional details with regard tointerlacing data in streaming image data will be discussed below withreference to FIGS. 7-9 .

By way of introduction, FIG. 1 illustrates an example industrialautomation system 10 controlled by one or more industrial controlsystems 11 and monitored by users 50 (e.g., plant operators from acontrol room 52) via a data streaming device 12. The present embodimentsdescribed herein may be implemented using the various equipment,devices, and machines illustrated in the industrial automation system 10described below. However, it should be noted that although the exampleindustrial automation system 10 of FIG. 1 is directed at a beveragepackaging plant, the present embodiments described herein may beemployed within any suitable industry or industrial segments, such asprocessing, refining, automotive, mining, power generation, hydrocarbonproduction, manufacturing, and the like. The following brief descriptionof the example industrial automation system 10 employed by the beveragepackaging plant is provided herein to help facilitate a morecomprehensive understanding of how the embodiments described herein maybe applied to industrial equipment, devices, and machines tosignificantly improve the operations of the respective industrialautomation system. As such, the embodiments described herein should notbe limited to be applied to the example depicted in FIG. 1 .

As depicted, the industrial automation system 10 includes stationshaving equipment, devices, machines, and/or machine components toconduct a particular application within an automated process, forexample, a beverage packaging process. The automated process may beginat a station 13 used for loading objects, such as empty cans or bottlesto be filled, into the industrial automation system 10 via a conveyorsection 14. The conveyor section 14 may transport the objects to astation 16 to perform a first action, for example, washing the emptycans and/or bottles. As objects exit from the station 16, the conveyorsection 14 may transport the objects to a station 20, such as a fillingand sealing station, in a single-file line. A second conveyor section 14may transport objects from the station 20 to a station 26. After theobjects proceed through the various stations, the objects may be removedfrom the station 28, for example, for storage in a warehouse 30.

It should be noted that, for different applications, the particularsystem, equipment, devices, machines, and/or machine components, may bedifferent or specially adapted to the respective application. In someembodiments, the industrial automation system 10 may include a varietyof equipment that may perform various operations as part of anindustrial application. For example, industrial automation system 10 mayinclude electrical equipment (e.g., electric motors, generators, fans,transformers, or regulators), hydraulic equipment (e.g., hydraulicpumps, tanks), compressed air equipment (e.g., compressors, condensers),steam equipment (e.g., boilers), mechanical equipment (e.g., lathes,milling machines), protective equipment, refrigeration equipment (e.g.,coolers), heating equipment (e.g., heaters), combustion equipment (e.g.,engines), power lines, hydraulic lines, steam lines, and the like. Someexample types of equipment may include mixers, machine conveyors, tanks,skids, specialized original equipment manufacturer machines, and thelike.

In some embodiments, the industrial automation system 10 may include avariety of machines and devices to perform various operations in acompressor station, an oil refinery, a batch operation for making fooditems, a mechanized assembly line, and so forth. Accordingly, theindustrial automation system 10 may comprise a variety of operationaldevices, machines or machine components, such as valves, actuators,meters, panels, network devices (e.g., switches, hubs, routers,gateways, modems), computing devices (e.g., microprocessor computers),communication devices (e.g., transceivers, transmitters, receivers),wheels, axles, shafts, wedges, joints, or a myriad of machinery ordevices used for manufacturing, processing, material handling, or othertypes of applications.

In some embodiments, the variety of equipment, machines, and devicesdescribed above may include industrial automation devices or componentsthat are used to automatically perform certain operations (e.g.,manufacturing, assembling, packaging, or testing). The industrialautomation devices or components may include various types of devices orcomponents that may be used to perform various operations that may bepart of an industrial application. For example, the industrialautomation devices or components may include electrical devices,hydraulic devices, compressed air devices, steam devices, mechanicaltools, protective devices, refrigeration devices, power lines, hydrauliclines, steam lines, and the like. Some example types of the industrialautomation devices or components may include industrial robots,automation cells, conveyors, lifters, turn-over machines, mixers, tanks,skids, specialized original equipment manufacturer machines,input/output (I/O) modules, motors, human machine interfaces (HMIs),operator interfaces, contactors, starters, actuators, drives, relays,protection devices, switchgear, compressors, firewall, network switches(e.g., Ethernet switches, modular-managed, fixed-managed,service-router, industrial, unmanaged, etc.), and the like.

In some cases, the industrial automation devices or components may becommunicatively coupled to monitoring or controlling devices that maymonitor or control the operations of the respective industrialautomation devices or components. The monitoring or controlling devicesmay include, for examples, local controllers (e.g., motor drivecontrollers, robot controllers, conveyor controller, liftercontrollers), sensing devices (e.g., various sensors, gauges, flowmeters) that are capable of measuring different properties (temperature,pressure, sound, light, voltage, flow, current, stress, speed, position,or orientation) of the coupled industrial automation devices orcomponents. The industrial automation devices or components may receivedata (e.g., commands, instructions) from the coupled devices and performrespective operations based on the received data. For example, acontroller of a motor drive may receive temperature data regarding atemperature of a connected motor and may cause the motor drive to adjustoperations of the motor based on the temperature data.

The one or more industrial control systems 11 may be communicativelycoupled, for example, via the monitoring/controlling devices orcomponents described above, to the respective industrial automationdevices or components. One or more properties of the industrialautomation devices or components may be monitored (e.g., via the sensingdevices) and controlled (e.g., via the local controllers) by the one ormore industrial control systems 11 for regulating control parametersand/or variables. For example, the one or more industrial controlsystems 11 may use a variety of sensing devices (e.g., sensors 31)employed in different locations within the industrial automation system10 to monitor the one or more properties of the industrial automationdevices or components. The sensors 31 may include any suitable sensors,such as temperature sensors, pressure sensors, acoustic sensors, opticalsensors, voltage sensors, current sensors, stress sensors, speed orvelocity sensors, position sensors, orientation sensors, and so forth.

The one or more industrial control systems 11 may use the localcontrollers (e.g., robot controllers, conveyor controller, or liftercontrollers) to control operations of the industrial automation devicesor components based on certain inputs, such as input from the sensor 31,input from onsite users 38 (e.g., maintenance personnel), or input fromremote users (e.g., the users 50). In one embodiment, the one or moreindustrial control systems 11 may receive sensing data regarding atemperature of an electrical motor from a temperature sensors associatedto the electrical motor. The one or more industrial control systems 11may send a command to a motor driver to adjust the motor speed based onthe temperature. In one embodiment, the one or more industrial controlsystems 11 may determine a fault associated with a conveyor based onpositions (e.g., horizontal and vertical positions) of the conveyormeasured by a Light Detection and Ranging (LiDAR) device and relevantacoustic measurement (e.g., noise pattern) of the conveyor measured byacoustic sensors. Based on the determined fault, the one or moreindustrial control systems 11 may cause a conveyor controller to suspendoperations of the conveyor for further diagnose or trouble shooting.Additionally, the one or more industrial control systems 11 may adjustcorresponding operations of other equipment, devices, or machines (e.g.,at the station 16 or the station 20) that may be related to theoperations of the conveyor.

In certain cases, audio devices 40 may be used to assist the onsiteusers 38 in remotely controlling or monitoring the industrial automationdevices or components (e.g., while located away from respective controlpanels of the local controllers or the control room 52) via verbal oraudible commands. The audio device 40 may perform actions in response toverbal or audible commands of the onsite users 38. For instance, theaudio devices 40 may interpret the audible commands, determinecorresponding commands for respective industrial automation devices orcomponents, and adjust the operations of the respective industrialautomation devices or components based on the determined commands. Inthis way, the onsite users 38 may physically input a change tooperations of a first machine via an interface of the first machine,while using the audio device 40 to change operations of a second machineusing verbal or audible commands without physically accessing the secondmachine. For example, the onsite users 38, while at the station 26 mayverbally request an audio device of the audio devices 40 to report on orchange an operation of the station 28 while physically located away froma control panel for the station 28.

The one or more industrial control systems 11 may include programmingobjects (e.g., software, firmware) that may be instantiated and executedto provide operation controls as described above, visualizationsrepresentative of live operational parameters of correspondingindustrial automation devices, and simulated functionalities (e.g., viadigital representations of the corresponding industrial automationdevices) similar or identical to the actual industrial automationdevices. The programming objects may include code and/or instructionsstored in memory circuitry of the one or more industrial control systems11 and executed by processing circuitry of the one or more industrialcontrol systems 11. The processing circuitry may communicate with thememory circuitry to permit a storage of relevant datasets (e.g., thevisualizations).

In some embodiments, the one or more industrial control systems 11 maygenerate the visualizations of representative of live operationalparameters of corresponding industrial automation devices based onsignals (e.g., analogue measurement) or datasets (e.g., digitalizedmeasurement) transmitted from the sensors 31. In some embodiments, thevisualizations of corresponding industrial automation devices may bebased on image data, such as images of control panels, dashboardvisualizations, operational visualizations, and other types ofvisualizations of the local controllers directly generated from thelocal controllers or indirectly generated from certain sensing devices(e.g., cameras) that are monitoring the control panels of the localcontrollers.

In some embodiments, the visualization of corresponding industrialautomation devices may be combined or enhanced (e.g., augmented) withother relevant visualizations (e.g., visualizations based on simulationsby digital representations of the corresponding industrial automationdevices). In some embodiments, the visualizations of correspondingindustrial automation devices may be generated based on holographicimages (e.g., three dimensional view combined with multi-dimensionalmeasurement) associated with the corresponding industrial automationdevices.

As mentioned above, the one or more industrial control systems 11 maygenerate datasets (e.g., image datasets) related to various aspects(e.g., operations, software, firmware, or maintenance) of the variety ofequipment, devices, and machines communicatively coupled to the one ormore industrial control systems 11. For example, the datasets mayinclude machine-readable images (e.g., Quick Response (QR) codes)containing information related to updates in operations, firmware, ormaintenance history associated with respective equipment, devices, andmachines. In some cases, the machine-readable images may includeadditional information of other equipment, devices, and machines thatmay be related to the respective equipment, devices, and machines (e.g.,implemented upstream and/or downstream with respect to the respectiveequipment, devices, and machines). In some embodiments, the one or moreindustrial control systems 11 may interlace the machine-readable imageswith streaming image data generated by the data streaming device 12. Theinterlaced data may be provided to the users 50 for view and to acomputing systems for data extraction and analysis.

The data streaming device 12 may be referred to as an electronic devicethat receives a various types of data (e.g., video, audio, logs) andstreams the data, such that user devices (e.g., computer monitors,screens, displays of mobile devices, smart TVs) may present the data viaa live streaming service. The data streaming device 12 may be connectedto one or more networks (e.g., intranet, internet) through variousconnections, such as Ethernet cables and/or Wi-Fi access points. Users(e.g., the users 50) may access streaming data via the user devices withcertain agreements (e.g., publisher-subscriber agreements). Suchagreements may include certain security protocols (e.g., governing dataaccess rights) to enforce data privacy and/or data security.

The term “streaming” used herein may be referred to as continuous (e.g.,with no beginning or end) data streams that provide a constant feed ofdata allowing the users to utilize live streaming data without delays(e.g., delays caused by data downloading). The live streaming data maybe created by streaming devices (e.g., the data streaming device 12)using data generated by various sources (e.g., industrial automationdevices, local controllers, sensing devices) in various formats (e.g.,audio, video, log files) and volumes. That is, the streaming devices mayaggregate the data from the various sources into seamlessly real-timedata that allows the users to utilize and act upon the live streamingdata in a real time and secured manner. Moreover, the live streamingdata may not be stored in the industrial control systems 11, the datastreaming device 12, or the recipient devices to allow for efficient(e.g., reduced network bandwidth) presentation of the data.

In some embodiments, the one or more industrial control systems 11 andthe data streaming device 12 may be communicatively coupled to otherdevices that include one or more displays 54 (e.g., conventionaldisplays, augmented reality displays, holographic displays) and humanmachine interfaces (HMI) 56 (e.g., touchscreens, touchpads, keyboards)via a network 60. For example, the users 50 (e.g., plant operators) mayview the streaming image data via the one or more displays 54 located ina remote location (e.g., the control room 52), and may interact with(e.g., analyze) the streaming image data via the human machineinterfaces 56. The users 50 may use certain computing systems or devices(e.g., control consoles) to send commands, via the network 60, to thelocal controllers based on the streaming image data that indicatescertain changes to operating environment, the industrial automationdevices or components, process variables associated with the industrialautomation system 10, and the like. For example, the streaming imagedata may indicate that one or more operation parameters (e.g., motorspeed, valve pressure) are outside expected ranges (e.g., predeterminedspeed or pressure range based on historical data). Based on the receivedcommands, the local controllers may cause (e.g., by changing operationsettings or parameters) the connected industrial automation devices orcomponents (e.g., motor, valve) to adjust operations (e.g., change valvesetting to prevent over pressure) in response to the changes indicatedin the streaming image data. In certain cases, the users 50 maycoordinate with the onsite users 38. For example, the users 50 may sendverbal commands, via the audio devices 40, to the onsite users 38 tochange certain operation settings of equipment at one or more stations(e.g., station 26 or 28).

In some embodiments, one or more edge computing devices 70 may beemployed between the data streaming device 12 and the network 60. One ormore databases 72 may be connected to the one or more edge computingdevices 70 to facilitate data storage related to edge computing. Asmentioned previously, visualizations of industrial automation devicesgenerated by the one or more control systems 11 may be combined orenhanced (e.g., augmented) with other relevant visualizations, such asvisualizations based on simulations by digital representations (e.g.digital avatars) of corresponding industrial automation devices. In somecases, the digital representations may be locally installed in the oneor more control systems 11. While in other cases, the digitalrepresentations may be remotely installed in the one or more edgecomputing devices 70, such that computing resources (e.g., processingcircuitry, memory circuitry) of the one or more control systems 11 canbe utilized more efficiently for operation control. Moreover, byperforming edge computing (e.g., simulations) at locations near datasources (e.g., industrial automation devices or components, localcontrollers, sensing devices), use of the one or more edge computingdevices 70 may reduce an amount of data to be processed in a site (e.g.,remote data center, cloud). Additionally, or alternatively, the one ormore edge computing devices 70 may transmit data between a local network(e.g., network including the industrial automation system 10, the one ormore control systems 11, and the data streaming device 12) and a cloud(e.g., network 60). The one or more edge computing devices 70 maytranslate protocols or languages associated with data and used by localsystems or devices into protocols or languages used by the cloud wherethe data may be further processed.

In some embodiments, one or more computing systems 80 and one or moredatabases 82 may be used to analyze and process the live streaming data.The one or more computing systems 80 may include computing systemsoperated by an organization (e.g., a manufacturing plant) and additionalcomputing systems operated by one or more additional entities (e.g.,contractors). The one or more computing systems 80 may include anysuitable computing devices (e.g., servers) equipped with variousprocessors, storage devices, input/output devices, displaying devices,communication devices, software related to data analysis, processing,simulation, or the like.

The one or more computing systems 80 may be cloud-based computingsystems. For example, at least a portion of the one or more computingsystems 80 and/or the one or more databases 82 may be connected to othercomputing systems and/or databases (e.g., within the network 60) to forma larger scale computing network to perform cloud-based computing, suchas large-scale modeling or simulations that may use a vast amount ofdata collected from multiple industrial systems. Such cloud-basedcomputing may provide insightful information (e.g., predictions) for alarge-scale industrial operation (e.g., a complete industrialsupply-manufacturing-inspection-shipping chain) associated with anorganization.

Although one data streaming device 12 is depicted, it should beunderstood that any suitable number of data streaming devices may beused in a particular industrial automation system 10 embodiment. Forinstance, multiple data streaming devices may be implemented in alarge-scale industrial operation (e.g., a complete industrialsupply-manufacturing-inspection-shipping chain) associated with anorganization, including a supply system, a manufacturing system, aninspection system, a shipping system, and so on. For each system of thelarge-scale industrial operation, one or more data streaming devices maybe used the enable multiple users remotely monitor various aspects(e.g., operations, software, firmware, or maintenance) associated withindividual systems, adjust corresponding operations based on themonitored data (e.g., live streaming data), coordinate certainoperations related to cooperation(s) between different systems, and thelike.

In certain cases, industrial data, such as image data associated withindustrial equipment and devices, may be provided to displays (e.g.,electronic displays, meters, or panels) present on the industrialequipment or devices located in certain locations (e.g., plant floors).The industrial data may include visualizations that are not presented onother display located in remote areas away from the certain locationshaving the equipment or devices installed. In such cases, the industrialdata may be provided to a live-streaming device or component, such thatthe industrial data may be pushed to users in the remote areas via adigital channel as a live-streaming broadcast.

The live-streaming device or component, such as the data streamingdevice 12, may be any type of electronic device capable of aggregatingvarious data (e.g., video, audio, logs) into a processing unit (e.g.,processor) and seamlessly streaming the data (e.g., via live-streambroadcast) onto remote user devices (e.g., monitors, screens, displays,TVs). Users (e.g., the users 50) may access live streaming data via theuser devices with certain agreements (e.g., publisher-subscriberagreements) containing one or more security protocols (e.g., related touser access rights to the live streaming data). As such, the users mayutilize the live streaming data without delays (e.g., delays caused bydata downloading), therefore acting upon the live streaming data in areal time and secured manner.

The data streaming device 12 may provide a variety of functions thatinclude, but are not limited to, accessing one or more industrialsystems (e.g., the one or more industrial control systems 11), one ormore devices (e.g., e.g., the one or more edge computing devices 70),and one or more networks (e.g., the network 60), or the like;selectively receiving data including image data containing multiplevisualizations each representative of one or more live operationalparameters of one or more devices in an industrial system (e.g., theindustrial automation system 10); and aggregating the data from selecteddata sources into seamlessly live streaming data; pushing (e.g., livebroadcasting) the live streaming data to users in a real time andsecured manner.

By way of example, FIG. 2 illustrates example components of the datastreaming device 12 of FIG. 1 . The data streaming device 12 may includevarious types of components that may assist the data streaming device 12in performing various types of live data streaming tasks. For example,the data streaming device 12 may include a communication component 102,a processor 104, a memory 106, a storage 108, input/output (I/O) ports110, a display 112, and the like.

The communication component 102 may be a wireless or wired communicationcomponent that may facilitate communication between the data streamingdevice 12 and various other systems (e.g., the one or more industrialcontrol systems 11), devices (e.g., the one or more edge computingdevices 70), networks (e.g., the network 60), or the like. For example,the communication component 102 may allow the data streaming device 12to receive live data from the one or more industrial control systems 11,such as image data, audio data, and log data representative of variousmeasurement values (e.g., positions, orientations, speeds, temperatures,noise levels, pressures, voltages, currents), operation values (e.g.,parameters, settings), and operation status (e.g., service logs, statusreports, incident reports, warnings, or alerts) associated withoperations of the various industrial equipment and devices in theindustrial automation system 10. The communication component 102 mayreceive and send notifications to the user devices (e.g., the one ormore displays 54, the human machine interface 56) and other computingsystems (e.g., the one or more computing systems 80). The communicationcomponent 102 may use a variety of communication protocols, such as OpenDatabase Connectivity (ODBC), TCP/IP Protocol, Distributed RelationalDatabase Architecture (DRDA) protocol, Database Change Protocol (DCP),HTTP protocol, other suitable current or future protocols, orcombinations thereof.

The processor 104 may process instructions for execution within the datastreaming device 12. The processor 104 may include single-threadedprocessor(s), multi-threaded processor(s), or both. The processor 104may process instructions stored in the memory 106. The processor 104 mayalso include hardware-based processor(s) each including one or morecores. The processor 104 may include general purpose processor(s),special purpose processor(s), or both. The processor 104 may becommunicatively coupled to other internal components (such as thecommunication component 102, the storage 108, the I/O ports 110, and thedisplay 112).

The memory 106 and the storage 108 may be any suitable articles ofmanufacture that can serve as media to store processor-executable code,data, or the like.

These articles of manufacture may represent computer-readable media(e.g., any suitable form of memory or storage) that may store theprocessor-executable code used by the processor 104 to perform thepresently disclosed techniques. As used herein, applications may includeany suitable computer software or program that may be installed onto thedata streaming device 12 and executed by the processor 104. The memory106 and the storage 108 may represent non-transitory computer-readablemedia (e.g., any suitable form of memory or storage) that may store theprocessor-executable code used by the processor 104 to perform varioustechniques described herein. It should be noted that non-transitorymerely indicates that the media is tangible and not a signal.

The I/O ports 110 may be interfaces that may couple to other peripheralcomponents such as input devices (e.g., keyboard, mouse), sensors,input/output (I/O) modules, and the like. The display 112 may operate asa human machine interface (HMI) to depict visualizations associated withsoftware or executable code being processed by the processor 104. In oneembodiment, the display 112 may be a touch display capable of receivinginputs from an operator of the data streaming device 12. The display 112may be any suitable type of display, such as a liquid crystal display(LCD), plasma display, or an organic light emitting diode (OLED)display, for example. Additionally, in one embodiment, the display 112may be provided in conjunction with a touch-sensitive mechanism (e.g., atouch screen) that may function as part of a control interface for thedata streaming device 12.

It should be noted that the components described above with regard tothe data streaming device 12 are examples and the data streaming device12 may include additional or fewer components relative to theillustrated embodiment. For example, the data streaming device 12 mayinclude additional circuitry that may include artificial intelligencecircuitry (e.g., neural network circuitry) for provide additionalcapabilities (e.g., advanced learnings and/or simulations to facilitatedata communication, selection, and aggregation).

As mentioned above, in some embodiments, the one or more industrialcontrol systems 11 may monitor and collect live data representative ofcertain measurement values, operation values, and operation statusesassociated with operations of the various industrial equipment anddevices. The one or more industrial control systems 11 may transmit thecollected live data to the data streaming device 12 that may providelive data streams associated with operations of the various industrialequipment and devices. The live data streams may enable users (e.g.,users 50) to interact with user devices (e.g., the one or more displays54, the human machine interface 56) and other computing systems (e.g.,the one or more computing systems 80) to access certain live datasetsvia the live data streams. In this way, the users may utilize the livestreams without delays and be able to act upon the live data streams ina real time manner.

With the foregoing in mind, FIG. 3 illustrates a flow chart of anexample method 150 for providing live streaming visualizations using thedata streaming device 12 of FIG. 1 . The data streaming device 12 mayperform operations described below via the processor 104 based onprocessor-executable code stored in the memory 106 and the storage 108.The processor 104 may execute the processor-executable code to performoperations, such as accessing the one or more industrial control systems11, the one or more edge computing devices 70, and the network 60;selectively receiving data containing visualizations representative oflive operational parameters or other characteristics of one or moredevices in the industrial automation system 10; aggregating the receiveddata into seamlessly live streaming data; and broadcasting the livestreaming data to users in a real time and secured manner.

Although the method 150 described in FIG. 3 is described in a particularorder, it should be noted that the method 150 may be performed in anysuitable order and is not limited to the order presented herein. Itshould also be noted that although each processing block is describedbelow in the method 150 as being performed by the data streaming device12, other suitable devices may perform the methods described herein.

Referring now to FIG. 3 , at block 152, the data streaming device 12 mayreceive a request containing credentials from a computing system. Forexample, a user (e.g., plant operator from the control room 52) may senda request for visualizations that corresponds to respective operationsof certain industrial equipment and devices. The user may send therequest via a user device (the human machine interface 56) that mayinteract with the computing system (e.g., one of the one or morecomputing systems 80), the edge computing devices 70, the data streamingdevice 12, the industrial control system 11, or the like. The requestmay include the credentials, such as user credentials (e.g., username,user identification number, facial image data, biometric data, and thelike). In some embodiments, the credentials may be related to accountinformation, password data, authentication data, and other types of datato determine view access rights associated with the credentials. Thecredentials may also provide subscription information that details theview access rights available to the user. In some embodiments, thecredentials may include additional data, such as system or deviceidentification data (e.g., MAC address and/or IP address of thecomputing system or the user device). Such credentials may facilitatethe data streaming device 12 to authenticate the user sending therequest. Indeed, the credentials may be provided or stored in a memoryof the data streaming device 12 and may detail different types ofvisualizations, datasets, and other information that each respectiveuser may be able to access and view. In this way, information pertainingto the industrial automation system 10 may be limited to thoseindividuals that may have been vetted or authorized to view such data.

The data streaming device 12 may receive the request containing thecredentials from the computing system. Based on the receivedcredentials, at block 154, the data streaming device 12 may determinewhether the credentials are authenticated. For example, the datastreaming device 12 may query, via the processor 104, a database storedin the storage 108, for user authentication data (e.g., usernames,passwords, and user identification numbers pre-authenticated and storedin the database). The data streaming device 12 may determine whether thereceived credentials match with the user authentication data in thedatabase. Additionally, or alternatively, the data streaming device 12may examine the additional data (e.g., system or device identificationdata) included in the received credentials by querying the database todetermine whether the additional data match with system or devicerelated authentication data pre-authenticated and stored in thedatabase. Based on the determinations described above, the datastreaming device 12 may determine whether the credentials areauthenticated.

After the credentials are authenticated, at block 156, the datastreaming device 12 may determine view access rights based on theauthenticated credentials. For example, the data streaming device 12 maydetermine the view access rights associated with the user by queryingthe database stored in the storage 108 for access right data based onthe authenticated credentials. The access right data may specify type(s)of data (e.g., image data, audio data) accessible by the user, contentsof data (e.g., limited data contents, devices, visualizations, etc.based on the user credentials) accessible by the user, and the like. Thedata streaming device 12 may determine the view access rights based onthe access right data and the request. For instance, the user may sendthe request for accessing plant floor datasets associated withvisualizations corresponding to operations of the station 16 (e.g.,washing station) and the station 20 (e.g., filling and sealing station)or a portion of the industrial automation system 10. Afterauthenticating the credentials of the user, the data streaming device 12may determine that the view access rights associated with the user islimited to visualizations corresponding to operations of the station 20.In some embodiments, the authenticated credentials may providesubscription information that details the view access rights availableto the user. In such cases, the data streaming device 12 may determinethe view access rights based on the subscription information.

Using the same example described above, at block 158, the data streamingdevice 12 may send a list of devices associated with the access rightsto the user. For example, the list of devices may include devicesassociated with the station 20 but excludes devices associated with thestation 16 based on the determined view access rights of the user, asdescribed at block 156. In this way, visualizations associated withdevices on the plant floor may or may not be the same visualizationslive-streamed to the user.

At block 160, the data streaming device 12 may receive a selection of adevice from the user. Still using the same example described above, theuser may select a bottle sealing machine at the station 20 from the listof devices associated with the access rights sent by the data streamingdevice 12. The user may send the selection of the bottle sealing machineto the data streaming device 12 via some user device. Additionally oralternatively, after viewing the list of devices associated with theaccess rights, the user may also send an additional request withadditional credentials (e.g. an authorization with a supervisoridentification) that may be used by the data streaming device 12 todetermine additional view access rights using the operations describedabove (e.g., at blocks 154 and 156). If the additional credentials areauthenticated, the data streaming device 12 may send an additional listof devices (e.g., a bottle washing machine at station 16) to the userfor a new selection.

Based on received user selections (e.g., selecting a specific device,and/or selecting a particular visualization associated with the specificdevice), at block 162, the data streaming device 12 may send a list ofvisualizations associated with the device to the user. The user mayreview the list of visualizations and determine a visualization. Theuser may then send a selection of the determined visualizationassociated with the device to the data streaming device 12.

At block 164, the data streaming device 12 may receive the selection ofthe visualization from the user. Based on received selection of thevisualization, the data streaming device 12 may receive the selectedvisualization data from the selected device (e.g., via an industrialcontrol system governing the selected device) into seamlessly real-timedata (e.g., streaming image data). That is, the selection of thevisualization may correspond to a graphical user interfacevisualization, a dashboard visualization, an operational statusvisualization, real-time image data acquired by an image sensor (e.g.,camera, video camera), or other suitable image data that presentsinformation related to the equipment associated in the selectedvisualization.

At block 166, the data streaming device 12 may broadcast the selectedvisualization to the user device that transmitted the request. As such,the user may view the streaming image data in real time. The streamingimage data may allow the users to utilize and act upon the streamingimage data in a real time and secured manner without delay (e.g., causedby data downloading).

By providing governed view access rights to the streaming image datarepresentative of operational data of industrial equipment, the method150 described herein may be used in various industrial automationsystems to prevent substantive datasets from being transmitted andstored via network devices, thereby reducing abilities of unauthorizedusers (e.g., hackers) from retrieving or intercepting the streamingimage data.

With the preceding in mind, and to provide further familiarity with themethod described above, FIG. 4 illustrates example live streamingvisualizations 200 based on the method 150 of FIG. 3 . The livestreaming visualizations 200 may include operation status of equipmenton a plant floor of the industrial automation system 10. A user (e.g.,plant operator from the control room 52) may send a request withcredentials for accessing certain plant floor datasets associated withvisualizations corresponding to operations of equipment at the station16 (e.g., washing station). After authenticating the credentials of theuser and receiving user selections (e.g., a first selection indicatingthat water tanks 202, 212, 222, and 232 are interested, a secondselection indicating that visualizations of temperature and pressurevalues associated with the selected water tanks are interested), thedata streaming device 12 may route streaming image data associated withthe selected visualizations to the user device (e.g., from the one ormore computing systems 80). The visualizations may mirror thevisualizations presented via the industrial control systems 11 that maybe on site or they may be visualizations that are not currently beingpresented by the industrial control systems 11 but are prepared by theindustrial control systems 11. The streaming image data may allow theusers to utilize and act upon the streaming image data in a real timeand secured manner without delay (e.g., delay caused by datadownloading).

For instance, by observing the streaming image data, the user may noticethat a visualization 234 representative of real-time tank 232temperature value shows the temperature value of water in the water tank232 is 40 degree that is above a threshold value (e.g., 10 degree). Theuser may record time information related to the water tank 232 showingabnormal temperature value based on a visualization 242 representativeof current date and time. The user may then check water temperatures ofthe other water tanks, such as the water tanks 202 and 222, by observinga visualization 204 representative of real-time tank 202 temperaturevalue and a visualization 224 representative of real-time tank 222temperature value. Based on the visualizations, the user may determinethat the water temperature in water tank 232 is higher than the watertemperatures (e.g., 5 degree) in other water tanks (e.g., water tanks202 and 222) at the station 16.

Additionally, or alternatively, the user may check water pressures ofcertain water tanks (e.g., the water tanks 202, 212, and 232). Forexample, by selecting an input 244 on the human machine interface 56presenting the example live streaming visualizations 200, the user maybe directed to a second display (e.g., display 2) to observevisualizations 206, 216, and 236 representative of real-time waterpressure values of the corresponding water tanks 202, 212, and 232.Based on the visualizations on the second display, the user maydetermine that the water pressure value in water tank 232 is rising withrespect to time while the water pressure values in the water tanks 202and 212 remain stable with respect to time. The visualizationsassociated with both the temperature values and pressure values indifferent water tanks may provide confidence for the user to makefurther decisions related to the operations at the station 16. Forexample, the user may use the one or more computing systems 80 to send acommand to cause a local controller connected to the water tank 232 tosuspend operations related to the water tank 232. The other water tanks202, 212, and 222 may continue operations while the water tank 232 issuspended for further actions (e.g., diagnose, fixing).

In this illustrated example, the live streaming visualizations 200enabled by the live streaming image data routed by the data streamingdevice 12 allows the user to remotely monitor the operations at thestation 16 on the plant floor of the industrial automation system 10.Such monitored operations include real-time water temperature values andwater pressure values of the water tanks 202, 212, 222, and 232 that maychange over a time period. The changes may prompt the user to takefurther actions, such as additional monitoring, suspending correspondingequipment, diagnosing or troubleshooting, replacing or repairingcorresponding equipment, and the like. The data presented in the livestreaming visualizations 200 is pure image data (e.g., pictures)containing no real values associated with the represented equipment. Assuch, when the live streaming visualizations 200 is accessed byunauthorized personnel (e.g., hackers) during an unexpected event (e.g.,a security bridge), it is difficult for the unauthorized personnel tointerpret the image data and glean or systematically (e.g.,automatically) collect information related to the values presented inthe image data. That is, the live streaming visualizations described inpresent disclosure provide data security that allows the users ofindustrial automation systems to utilize and act upon the live streamingdata in a secured and real time manner.

In some cases, multiple users may be interested in viewing image dataassociated with the real time operation status of equipment on the plantfloor of the industrial automation system 10. By presenting livestreaming visualizations (e.g., similar to or different from the livestreaming visualizations 200 depending on user credentials) via the datastreaming device 12, each user of the multiple users may avoid gainingaccess to a respective port of a control system or device that outputsthe image data, thereby reducing the network latency and networkbandwidth provided to each user. Instead, any suitable number ofindividual users may view the image data without affecting the networklatency or available bandwidth of the control system or device.

In some embodiments, a computing system may extract historical data fromreceived streaming data based on certain identified data fields. Thestreaming data may include image data (e.g., image frames) and audiodata (e.g., audio packets). The computing system may associate the audiopackets with the image frames and extract one or more portions of theimage frames based on the identified data fields. Such data fields mayvary over a time period and be identified by using, for example, opticalcharacter recognition (OCR) technology. The computing system may acquiredata field values based on the extracted portions of the image framesand identify audio packets corresponding to the image frames associatedwith the extracted portions. The acquired data field values and theidentified audio packets may be stored as time series data in adatabase.

With the preceding in mind, FIG. 5 illustrates a flow chart of anexample method 250 for extracting historical data from the livestreaming data. A computing system (e.g., from the one or more computingsystems 80) may perform operations described below via one or moreprocessors based on processor-executable code stored in one or morememory devices and one or more storage devices. The one or moreprocessors may execute the processor-executable code to performoperations, such as associating audio packets with image frames,identifying certain data fields, extracting portions of the image framesbased on the identified data fields, acquiring data field values basedon the extracted portions, identifying audio packets corresponding tothe extracted image frames, and storing the acquired data field valuesand identified audio packets as time series data in a database.

Although the method 250 described in FIG. 5 is described in a particularorder, it should be noted that the method 250 may be performed in anysuitable order and is not limited to the order presented herein. Itshould also be noted that although each processing block is describedbelow in the method 250 as being performed by the computing system fromthe one or more computing systems 80, other suitable computing systemsor devices (e.g., systems or devices connected to the network 60) mayperform the methods described herein.

Returning to FIG. 5 , at block 252, the computing system may receive thestreaming data. The streaming data, as discussed above, may betransmitted via a streaming device (e.g., the data streaming device 12)using data generated by various sources (e.g., industrial automationdevices, controllers, sensing devices) in various formats and volumes.In some embodiments, the streaming data may include audio data that isassociated (e.g., synchronized) with image data in the streaming data.For example, a camera (e.g., a camera with audio sensors) may bepositioned in front of a system or device (e.g., control system, controlpanel, door assembly, or the like). The image data and audio dataassociated with the system or device and acquired by the camera may bemade available via the live-streaming device for view by users. In someembodiments, the image data and the audio data may be acquired bydifferent devices (e.g., a camera and an audio sensing device). Thecomputing system may receive the image data and the audio data are viacertain data ports and may use the same amount of bandwidth regardlessof whether the audio data is present or not.

In some embodiments, the image data may include image frames. Forexample, the image data may be broadcast at a refresh rate (e.g., 60,70, 80, or 100 Hz), such that during each second that the image data isdisplayed, a number (e.g., 60, 70, 80, or 100) of image frames arepresented. The individual image frames may not be perceivable by theuser (e.g., via human eye) at the refresh rates used by the display andtransmitted by the streaming device. However, a viewing device (e.g.,computer, tablet) displaying the streaming image data may be capable ofdetecting certain features (e.g., interlaced machine-readable images)that may be embedded between the image frames using certain technologies(e.g., sampling the streaming image data at a specific rate comparableto the refresh rate).

In some embodiments, the audio data may include audio packets. The audiopackets in the audio data may be associated with the image frames in theimage data. For example, a camera positioned in front of the water tank202 of FIG. 4 may acquire image frames based on a pressure display panelof the water tank 202. An acoustic sensor positioned close to an outletvalve of the water tank 202 may acquire audio packets based on acousticwaves arriving at the acoustic sensor. The arriving acoustic waves mayinclude sound profiles of the water tank 202, voices within the vicinityof the water tank 202, or the like. When the pressure of the water tank202 increases (e.g., due to increased water temperature inside the watertank 202), a noise associated with the increase water tank pressure andpresented in the acoustic waves may be acquired by the acoustic sensorand a corresponding noise pattern may be recorded in certain audiopackets.

In some embodiments, the image data and the audio data may besynchronized. For example, each of the image frames may be assigned witha time tag or stamp (e.g., containing the time when the image frame isacquired), and each corresponding audio packet may be assigned with thesame time tag during broadcasting of the streaming data. For a differentexample, each of the image frames may be assigned with a unique number(e.g., serial number or frame number related to the time when the imageframe is acquired), and each corresponding audio packet may be assignedwith the same unique number during the broadcasting. Some identifiers(e.g., text, number related to time tag/stamp) may be presented inpictures of equipment display panels and other identifies (e.g., imageframe number) may be superimposed on images (e.g., broadcasted streamingimage data).

After receiving the streaming data, at block 254, the computing systemmay associate the audio packets of the streaming data with the imageframes of the streaming data. In some embodiments, the computing systemmay associate the audio packets with the corresponding image framesbased on the time tags assigned to the audio packets and the imageframes. In some embodiments, the computing system may associate theaudio packets with the corresponding image frames based on uniquenumbers assigned to the audio packets and the image frames.

At block 256, the computing system may identify one or more data fieldsin the image frames. For example, by monitoring the streaming data, suchas the live streaming visualizations 200 of FIG. 4 that may includeoperation status of water tanks at station 16, a user (e.g., plantoperator) may notice a type of noise associated with the water tank 232being presented in the audio data of the streaming data. The user maysend a command to the computing system to analyze portions of the imageframes associated with the water tank 232, such as the portions of theimage frames that include a tank temperature display panel. By analyzingthe image frames over a time period (e.g., from a predetermined time,such as 3 minute, 5 minutes, or 10 minutes, prior to the time receivingthe command to the present time), the computing system may track thetank temperature value and and determine that the tank pressure value isincreasing in a particular time window (e.g., from 2 minutes prior tothe commanding time to the present time), indicating a potential issueoccurred to the water tank 232.

In addition to using user input, in some embodiments, the computingsystem may monitor the received image data and detect portions of theimage data that changes. That is, in a display panel visualization, theportions of the image data between variable data values may remain thesame for a certain period of time, while other portions of the imagedata may change due to changing values or measurements corresponding tostatus updates or changes.

After identifying the one or more data fields, at block 258, thecomputing system may extract one or more portions of the image framesbased on the data fields over a time period. For example, based on thedata analysis at block 256, the computing system may extract a portionof the image data associated with measurements related to the water tank232 within the particular time window indicating the rising tanktemperature and tank pressure.

At block 260, the computing system may acquire one or more data fieldvalues from the extracted portions of the image data. In someembodiments, the computing system may use the extracted portions in thetime period and corresponding reference data to acquire the one or moredata field values. For example, the computing system may use thereference data (e.g., specifications and settings of the water tank 232,pressure sensors and temperature sensors connected to the water tank232) to derive or interpret real values (e.g., temperature value,pressure value) corresponding to the extracted images. As mentionedpreviously, the image data of streaming data contains image data thatexcludes substantive data or measurements (e.g., temperature or pressurereading) of operation-related values. Therefore, even if the image datais compromised (e.g., hacked), without the knowledge (e.g., thereference data) of the corresponding equipment or devices, it isdifficult for unauthorized personnel to access the substantive valuespresented (e.g. displayed) in the image data.

In some embodiments, the computing system may include an opticalcharacter recognition (OCR) tool that may identify the one or more datafield values from the streaming data and store the identified fieldvalues in a time series database. For example, the OCR tool may becapable of converting images of text (e.g., presented in pictures ofequipment display panels) and/or subtitle text superimposed on images(e.g., broadcasted streaming image data) into machine-encoded text.

The identified field values may be captured at various intervals and mayinclude an associated frame number or time stamp that provides temporalcontext to the data values. The data capture process may enable theunstructured data (e.g., image data) presented in the images to becomestructured data (e.g., time series data). In some embodiments, thecomputing system may include an artificial intelligence or machinelearning module that may learn (e.g., based on historical data) toidentify data fields in different visualizations provided by differentsources (e.g., automation devices, controllers, or sensing devices) ofthe streaming data.

At block 262, the computing system may identify audio packetscorresponding to the image frames associated with the field values. Forexample, the computing system may determine a time range associated withthe extracted portions of the image data based on the time tags assignedto the extracted portions. The computing system may then identify theaudio packets corresponding to the extracted portions based on the timetags assigned to the audio packets that are within the time range. Inanother example, the computing system may determine a unique numberrange (e.g., containing unique serial numbers or frame numbers)associated with the extracted portions based on the unique numbersassigned to the image frames of the extracted portions. The computingsystem may then identify the audio packets (e.g., over a period of time)corresponding to the extracted portions of the image data based on theunique numbers assigned to the audio packets that are within the uniquenumber range.

At block 264, the computing system may store the acquired data fieldvalues and identified audio packets in a time series database. Forinstance, the computing system may include an optical characterrecognition (OCR) tool that may acquire the data field values from thestreaming data, transform the acquired data field values into timeseries image datasets, and store the digital values in time seriesdatasets. The stored time series image may be utilized by the computingsystem or other computing systems or devices for analyzing, plotting(e.g., plotting temperature vs. time diagrams), programing (e.g.,designing a GUI showing modeling result based on pressure values), andthe like. Moreover, the audio data that corresponds to the data fieldvalues may also be stored in a database (e.g., separate or the same)with a pointer or some other reference tool associating the audio datawith corresponding data field values. In this way, the audio dataassociated with the acquired data field values may be accessible forplayback to better understand the data field values.

In some embodiments, image data from a separate image data source (e.g.,camera) may be received and associated with the acquired data fieldvalues. That is, a video camera or security camera acquiring image datarelated to the physical components being monitored in the visualizationsrepresented by the streaming data may be received by the computingsystem and associated with the acquired data fields as described above.In this way, image data related to the operations of equipmentassociated with the acquired data fields may be available for accessupon request.

In some embodiments, the computing system may apply certain dataprocessing techniques, such as filtering (e.g., bandpass filtering) andapplying a Fourier Transform, to acquired audio data (e.g., audiopackets) and store digital representations of audio data in thedatabase. In some embodiments, the identified audio packets may bestored along with the acquired data field values, such that the usersmay have access to sound profiles of certain events that occur prior towarnings or alarms associated with corresponding devices being reachedor instances. For example, an acoustic sensor positioned close to anoutlet valve of a water tank may acquire audio packets based on acousticwaves emitted from the water tank. A camera positioned in front of thewater tank may acquire image data based on a pressure display panel ofthe water tank, and the OCR tool may be used to acquire the data fieldvalues (e.g., tank pressure values) based on the image data. Theacquired audio packets and data field values may be stored as timeseries audio datasets and time series datasets (e.g., as digitalvalues), respectively. By storing the datasets as described above, theusers may have access to sound profiles of certain events (e.g., noisepatterns associated with increasing water tank pressure) thatcorresponds to changes in data that may be tracked or detected based ontrend analysis or other analysis performed on the acquired data fieldvalues. Such events may occur prior to warnings or alarms (e.g., waterpressure being above a threshold pressure value) associated withcorresponding devices (e.g., water tank) being reached or instances. Theidentified audio packets with the corresponding acquired data fieldvalues may provide precautionary information for enabling preventiveoperations. For example, based on the noise patterns and the acquiredwater tank pressure value, a plant operator or the computing system maysend a command to a water tank controller to suspend operations of awater tank having the noise patterns and increasing water pressure. Suchoperations may prevent the water tank from being over pressure.

By providing structured data (e.g., time series datasets), the method250 described herein may be used in various industrial automationsystems in which unstructured data (e.g., image data) presented via livestream visualizations may be captured and stored in a structured formatfor further data processing and analysis. For instance, the samecomputing system or a different computing system may receive time seriesdata from the data base and identify certain data trends associated withrespective equipment or devices based on received time series data. Thecomputing system may predict certain operational parameters of therespective equipment or devices based on identified trends. If apredicted operational parameter of an equipment is outside an expectedrange (e.g., predetermined range based on historical data), thecomputing system may send corresponding commands to the equipment toadjust corresponding operations.

With the foregoing in mind, FIG. 6 illustrates a flow chart of anexample method 300 for identifying data trends based on extractedhistorical data using the example method 250 of FIG. 5 . A computingsystem (e.g., from the one or more computing systems 80), which may ormay not be the same computing system used by the example method 250 ofFIG. 5 , may perform operations described below via one or moreprocessors based on processor-executable code stored in one or morememory devices and one or more storage devices. The one or moreprocessors may execute the processor-executable code to performoperations, such as receiving time series data from a data base,identifying data trends based on received time series data, predictingoperational parameters of based on identified trends, and sendingcommands to equipment to adjust corresponding operations whendetermining a predicted operational parameter of the equipment isoutside an expected range.

Although the method 300 described in FIG. 6 is described in a particularorder, it should be noted that the method 300 may be performed in anysuitable order and is not limited to the order presented herein. Itshould also be noted that although each processing block is describedbelow in the method 300 as being performed by the computing system fromthe one or more computing systems 80, other suitable computing systemsor devices (e.g., system or devices connected to the network 60) mayperform the methods described herein.

Returning to FIG. 6 , at block 302, the computing system may receivetime series data from a time series database. The time series data mayinclude image data, audio data, and other types of data representativeof live operational parameters or status of industrial devices. In someembodiments, the computing system may constantly receive time seriesdata associated with operations of a corresponding industrial automationsystem. In some embodiments, the computing system may periodically(e.g., based on a predetermined time interval, such as every 30 second,1, 3, 5, or 10 minutes) receive the time series data. In otherembodiments, the computing system may receive the time series data afterbeing trigged by certain events (e.g., increasing temperature detected,over pressure warning received, user request received).

After receiving the time series data from the time series database, thecomputing system may perform various types of analysis (e.g., trendanalysis, predictions) based on the time series data. For example, atblock 304, the computing system may identify one or more trendsassociated with one or more operation devices based on received timeseries data. In some embodiments, the computing system may utilizecertain hardware components (e.g., neural network-based machine learningcircuitry) and/or software components (e.g., analytic algorithms,machine learning algorithms) to identified the one or more trendsassociated with one or more operation devices.

For example, by analyzing the received time series data that includestime series data associated with a conveyor motor driving a conveyor ata convey section (e.g., the conveyor section 14 in the industrialautomation system 10), the computing system may identify a trendassociated with the conveyor motor. The time series data may be createdbased on image data of a display panel of a motion sensing device (e.g.,including motion sensors, Light Detection and Ranging (LiDAR) sensors)monitoring the conveyor motor. The motion sensing device may detectpositions and position changes of the conveyor motor with respect to abase where the conveyor motor sits. The values associated with thedetected positions and position changes may be displayed on the displaypanel. The visualizations of the display panel may be provided to a datastreaming device (e.g., data streaming device 12) via a controller(e.g., convey motor controller). The data values associated with thedetected positions may be acquired using the method 250.

In some embodiments, the time series audio data may include acousticwaves emitted from the conveyor motor. The computing system may convertthe detected acoustic waves into digital or data signals (e.g.,frequency domain). The signals may be analyzed to identify trends orother features that may be present in the data.

That is, in some embodiments, the computing system may identify aparticular noise pattern (or signature) in the received time seriesaudio data. For example, the computing system may use, for example,audio mode recognition and machine learning, to compare the soundprofile (e.g., via Fourier Transform) in the time series audio data toreference audio data stored in a database that contains different noisepatterns corresponding to different faulty operations of the conveyormotor identified in an operational history of the conveyor motoridentified and/or similar motors used elsewhere). Based on thecomparison, the computing system may determine that the particular noisepattern correspond to an offset of the conveyor motor. Next, thecomputing system may retrieve a portion of the time series data (e.g.,values) corresponding to the portion of time series audio data havingthe particular noise pattern. The retrieved portion of time series datamay include data that represent a position change (e.g., an offset) forthe conveyor motor with respect to the base. The computing system mayuse, for example, machine learning, to identify that an increasing trendassociated with the operation of the conveyor motor.

Based on the one or more identified trends, at block 306, the computingsystem may predict one or more future operational parameters of the oneor more devices based on the trends. In some embodiments, the computingsystem may utilize certain hardware components (e.g., e.g., artificialintelligence (AI) based processing circuitry) and/or software components(e.g., artificial intelligence algorithms) to predict the one or moreoperational parameters. Using the same example described above, based onthe identified trend indicating potential increasing offset values ofthe conveyor motor, the computing system may predict an offsetincreasing range (e.g., from 5 mm to 10 mm) within a time period (e.g.,next 1, 1.5, or 2 hours).

At block 308, the computing system may determine whether the predictedone or more operational parameters are outside one or more expectedranges. Using the same example as above, if the computing systemdetermines that the predicted offset value (e.g., 5 mm) within 1 hourmay be outside an expected range (e.g., from 2 mm to 4 mm, apredetermined range based on historical offset values measured by themotion sensing device), at block 310, the computing system may sendcommands to the one or more operation devices (e.g., the conveyor motorcontroller) to adjust corresponding operations (e.g., reducing motorspeed, or suspend the conveyor motor). In some embodiments, certainhistorical values (e.g., motor speed, water pressure) based onhistorical data (e.g., historical operation data associated with theindustrial devices) may be used to set alarm limits based on the averagevalues over a period of time. If an alarm limit has been exceeded, thecomputing device may send a notification (e.g., email, text message, andannunciation signal) to certain devices or components to alert one ormore users.

In certain embodiments, a control system (e.g., industrial controlsystem) may receive datasets from operation equipment or devices of anindustrial system. Based on the received datasets, the control systemmay generate machine-readable images (e.g., Quick Response (QR) codes,bar codes) representative of the datasets and interlace themachine-readable images with the streaming image data provided to usersfor view via a data streaming device. Such interlaced machine-readableimages may be detectable by certain computing devices but may not beperceivable by the user. In this way, additional information may beembedded within image frames of the streaming image data and provided tothe user via broadcast. In some cases, interlaced data may be used toprovide additional data while streaming the visualization images.

With the preceding in mind, FIG. 7 illustrates a flow chart of anexample method 350 for interlacing data in the live streaming data. Acontrol system (e.g., from the one or more industrial control systems11) may perform operations described below via one or more processorsbased on processor-executable code stored in one or more memory devicesand one or more storage devices. The one or more processors may executethe processor-executable code to perform operations, such as receivingstreaming data, querying the industrial devices for datasets associatedwith the streaming data, generating machine-readable images based on thedatasets, embedding the machine-readable images within the streamingdata, and sending the streaming data with embedded machine-readableimages to a data streaming device. In some embodiments, the operationsmay include interlacing a buffer of machine-readable images within thestreaming data (e.g., consisting of image frames) and retrieving amissing machine-readable image from one or more neighboring imageframes.

Although the method 350 described in FIG. 7 is described in a particularorder, it should be noted that the method 350 may be performed in anysuitable order and is not limited to the order presented herein. Itshould also be noted that although each processing block is describedbelow in the method 350 as being performed by the control system fromthe one or more industrial control systems 11, other suitable controlsystems or devices (e.g., system or devices connected to the industrialautomation system 10) may perform the methods described herein.

Returning to FIG. 7 , at block 352, a control system may receivestreaming data from industrial devices of an industrial system. Forexample, the control system may be one of the one or more industrialcontrol systems 11. The control system may generate visualizations eachrepresentative of live operational parameters associated with industrialdevices, determine access rights to each of the visualizations based ona request containing credentials associated with a requester, identifyone or more of the visualizations based on the access rights, obtain aselection of the one or more of the visualizations from the requester,and receive streaming data containing the selected visualizations, inaccordance with embodiments described above.

After receiving the streaming data, at block 354, the control system mayquery the industrial devices for datasets associated with the streamingdata. That is, the control system may query various data sources toidentify datasets that may be associated with one or more devicespresented via the streaming data. In some embodiments, the streamingdata may be associated with certain devices presented therein. That is,the streaming data may include metadata or associations to datasetsacquired by sensors or produced by other devices. In addition, a user orthe control system may define the datasets associated with the streamingdata.

For example, a motor (e.g., conveyor motor) may have an operation speedof 3000 revolutions per minute (RPM) and the streaming data may includea visualization indicative of a measurement of a motor speed sensor thatmay detect the current operation speed of the motor. In someembodiments, the motor speed sensor may be associated with the streamingdata as one of the data sources being represented by the streaming data.

In some embodiments, the control system generating the streaming datamay receive data from sensors being above some threshold. Here, thecontrol system may identify the corresponding datasets to embed with thestreaming data to convey the respective condition or data values.

With this in mind, in addition to generating and transmitting thestreaming data, the control system may query the correspondingindustrial devices for datasets in response to detecting certainwarnings or alerts. For example, some warnings or alerts may betriggered by detecting/identifying certain operational parametersassociated with the industrial devices are different from devicesettings, or above/below pre-determined threshold values or thresholdranges as described above. In some embodiments, the control system mayquery the corresponding industrial devices for datasets in response tocertain input, such as requests from users (e.g., plant operator),indications from other computing systems (e.g., computing systems 80)that may identify abnormal changes associated with operationalparameters of the industrial devices based on analyzing the streamingdata.

At block 356, the control system may generate machine-readable imagesbased on the datasets. The machine-readable images used herein may bereferred to as bar codes (machine-readable optical labels such as UPCcode), quick response (QR) codes (a type of matrix barcodes ortwo-dimensional barcode), or other suitable machine-readable image dataformats that may contain information related to items to which they areassociated (e.g., attached, linked). Such information may includelocation, identification, specification, status (e.g., operationparameters such as various device or sensor readings), other trackabledata, and the like. For example, a QR code may contain up to 7089characters of numerical data or 4296 alphanumerical characters that maybe used to store live operational parameters associated with industrialdevices. In some embodiments, the QR codes may use certain encodingmodes, such as numeric, alphanumeric, byte/binary, kanji, or otherextensions to store data efficiently.

After generating the machine-readable images, at block 358, the controlsystem may embed the machine-readable images within the streaming data.As mentioned above, image data may be provided at certain frequency,such as 60 Hz or higher. However, the human eye may only perceive imagesthat are presented at a lower frequency (e.g., 15 Hz). With this inmind, in some embodiments, datasets related to the operations of variousequipment or devices, such as the live operational parameters associatedwith industrial devices, may be interlaced with the image data streamedvia a live-streaming device (e.g., data streaming device 12). Forinstance, bar codes, QR codes, or some other machine-readable images maybe interlaced with the image data provided to an end user for view andanalyze. Such machine-readable images may include information related tochanges in operations, firmware, or maintenance history associated withrespective industrial equipment, devices, and machines. Additionally, oralternatively, the machine-readable images may include information ofother industrial equipment, devices, and machines that may be related(e.g., connected) to the respective equipment, devices, and machines.The interlaced machine-readable image may not be perceivable by a humanviewer, but the local computing system that displays the streamed imagedata may sample the streamed image data at a rate that enables it tocapture the interlaced machine-readable image data. In some embodiments,the interlaced machine-readable image may be encrypted, such that thelocal computing system that receives the streamed image data may haveaccess to a decryption tool to decrypt the machine-readable image.

In one embodiment, the image data associated with certain industrialdevices (e.g., water tanks water tanks 202, 212, 222, and 232 of FIG. 4) may be provided at 60 Hz frequency, such that the end user (e.g.,plant operator) may view and analyze 60 image frames per second. Thecontrol system may embed (e.g., superimpose) the machine-readable imageswithin the image frames. The control system may embed themachine-readable images with certain image frames based on apre-determined order. For instance, the control system may embed themachine-readable images with the 10th, 20th, 30th, 40th, 50th, and 60thimage frames within the 60 image frames of each second. In each of the10th, 20th, 30th, 40th, 50th, and 60th image frames, one or more (e.g.,three) machine-readable images may be embedded with the correspondingimage frames. Certain data redundancy (e.g., duplication or mirroring ofdata that helps prevent the data from missing or a device from becomingunavailable) may be added during a process of embedding themachine-readable images with the streaming data. For example, thecontrol system may create a high-density image that includes a number ofmachine-readable images at different positions during one frame of theimage data. Additional details with regard to interlacing data instreaming image data with added redundancy will be discussed below withreference to FIG. 8 .

At block 360, the control system may send the streaming data withembedded machine-readable images to a data streaming device. Forexample, the control system, such as one of the one or more industrialcontrol systems 11, may send the streaming data embedded with themachine-readable images to the data streaming device 12 via acommunication port. The data streaming device 12 may route the streamingdata to the end user for view and analyze.

FIG. 8 illustrates an example of a number of image frames withinterlaced data based on the example method 350 of FIG. 7 . A controlsystem (e.g., one of the one or more industrial control systems 11) or astreaming device (e.g., the data streaming device 12) may provide imagedata (e.g., the live streaming visualizations 200 of FIG. 4 ) associatedwith certain industrial devices (e.g., water tanks 202, 212, 222, and232 of FIG. 4 ) at 60 Hz frequency, such that the end user (e.g., plantoperator) may view and analyze 60 image frames per second. For example,the 60 image frames may include image frames 402 (FRAME 1), 404 (FRAME2), 406 (FRAME 10), 408 (FRAME 20), 410 (FRAME 30), 412 (FRAME 40), 414(FRAME 60), and other intermediate image frames not shown. Each imageframe may represent a live streaming visualization at a different timerepresentative of live operational parameters (e.g., water pressureand/or water temperature) associated with water tanks (e.g., the watertanks 202, 212, 222, and 232).

As mentioned above, the control system or the streaming device may embedthe machine-readable images (e.g., QR codes) with certain image framesbased on a pre-determined order. For instance, the control system or thestreaming device may embed the machine-readable images with the imageframes 406, 408, 410, and 420 within the 60 image frames as illustrated.In each of the image frames 406, 408, 410, and 420, threemachine-readable images may be embedded with the corresponding imageframe. With the live streaming image data interlaced themachine-readable images that include data values or information beingtransmitted by the control system. As such, the control system maycommunicate additional information (e.g., certain operationalparameters) associated with the industrial devices within the livestreaming image data.

Additionally or alternatively, additional data may be added to improvedata redundancy. In some cases, various network conditions may cause thestreaming image data to be missing certain frames of the image data. Assuch, if the interlaced machine-readable image is part of the missingframe, the local computing system may not receive the expected datasets.To better ensure that the streaming image data will provide the expecteddatasets even when the transmitting device may lose networkconnectivity, the live-streaming device or other suitable component maycreate a high-density image that includes a number of machine-readableimages at different positions during one frame of the image data. Thecollection of machine-readable images may correspond to a buffer ofdatasets. As the streaming image data continues to present image datavia each subsequent frame, the frames that include the interlacedmachine-readable images may move a new machine-readable image onto theframe and remove a previously presented machine-readable image off ofthe frame. As such, local computing system may receive a buffer ofdatasets via the machine-readable images, such that any missing framemay be identified based on how the machine-readable images cycle throughthe frames of image data. The missing frame may be accounted for andretrieved from a different frame.

For example, at the beginning of the 60 image frames, the image frame406 and 408 may be embedded with the first, second and thirdmachine-readable images 420, 422, and 424, such that certain informationrelated to events (e.g., changes associated with the live operationalparameters being detected at a beginning time) may be duplicated toprevent the information from missing (e.g., caused by networkconnectivity issues). As a subsequent image frame containing themachine-readable images, the image frame 410 may be embedded withdifferent machine-readable images 422, 424, and 426. For example, in theimage frame 410, the control system or the streaming device may move thefirst machine-readable image 420 out of the image frame 410 and push thefourth machine-readable image 426 into the image frame 410. As suchcycling process continues, the next subsequent image frame containingthe machine-readable images, the image frame 420 may be embedded withdifferent machine-readable images 424, 426, and 428.

In some embodiments, the control system or the streaming device mayinclude (e.g., superimposing) certain identifiers (e.g., code, image, ortext) in the image frames containing the machine-readable images. Forexample, the identifiers may include numerical characters (e.g., in apre-determined order) in each of the image frames containing themachine-readable images, such that a missing image frame containing themachine-readable images may be identified (e.g., by comparing theidentifiers of two consecutive image frames containing themachine-readable images to identify a missing intermediate identifierrepresenting a missing image frame).

Using the cycling process described above, certain data redundancy(e.g., duplication or mirroring of data that helps prevent the data frommissing or a device from becoming unavailable) may be added during theprocess of embedding the machine-readable images with the streamingdata. For instance, the beginning image frames containing themachine-readable images, such as the image frames 406 and 408, may beembedded with a same portion (e.g., the machine-readable images 422 and424) of the machine-readable images as the subsequent image frame 410containing the machine-readable images. Similarly, the image frame 410may be embedded with an identical portion (e.g., the machine-readableimages 424 and 426) of the machine-readable images as the subsequentimage frame 420. In this way, the control system or the streaming devicemay create a high-density image including a number of machine-readableimages at different positions during one frame of the image data.

Using interlacing data technique described above, a buffer of imagedatasets (e.g., a gather of high-density images consisting ofmachine-readable images) may be interlaced within the streaming imagedata, such that any missing image frames may be accounted for andretrieved from one or more neighboring image frames. In this way, thecontrol system or the streaming device may utilize the machine-readableimages interlaced with the streaming image data to ensure that thestreaming image data may provide expected image datasets to the userduring occurrences of various network issues.

For example, in present illustrated embodiment, the control system orthe streaming device may identify the image frame 410 as a missingframe. The missing frame 410 may be identified by the control system orthe streaming device by comparing identifiers of two consecutive imageframes containing the machine-readable images, such as the image frames408 and 420, to identify a missing intermediate identifier representingthe missing image frame 410. The control system or the streaming devicemay use one or more neighboring image frames to retrieve missingmachine-readable images associated with the missing image frame. Forexample, the control system or the streaming device may use the imageframes 408 and 420 to retrieve the missing machine-readable images 422,424, and 426 associate with the missing frame 410.

With the preceding in mind, FIG. 9 illustrates a flow chart of anexample method 450 for processing the live streaming data withinterlaced data. At block 452, a computing system (e.g., the computingsystems 80) may receive streaming data with embedded machine-readableimages. For example, the computing system may receive the streaming datawith embedded machine-readable images (e.g., bar codes, quick response(QR) codes, or other suitable machine-readable image data formats) froma streaming device (e.g., the data streaming device 12). The streamingdata may be provided as image frames by the streaming device at certainfrequency (e.g., 60 Hz or higher frequencies). At least a portion of theimage frames may be embedded with the machine-readable images asdescribed in previous embodiments.

After receiving the streaming data with embedded machine-readableimages, at block 454, the computing system may sample the image framesof received streaming data. for example, the computing system may samplethe image frames at a pre-determined frequency (e.g., 60 Hz or higherfrequencies) to retrieve individual image frames. Additionally oralternatively, the computing system may sample the image frames at adifferent pre-determined frequency (e.g., based on a per-determinedorder). For example, a control system or a streaming device may embedthe machine-readable images (e.g. QR codes) with certain image framesbased on the pre-determined order. Using the example described in FIG. 8, the control system or the streaming device may embed themachine-readable images (e.g., the machine-readable images 420, 422,424, 426, and 428) with the image frames 406, 408, 410, and 420 withinthe 60 image frames in a given time period (e.g., one second). In eachof the image frames 406, 408, 410, and 420, three machine-readableimages may be embedded with the corresponding image frame.

At block 456, the computing system may extract machine-readable imagesfrom received streaming data. For example, the computing system may usethe sampled image frames (from block 454) to extract themachine-readable images embedded in the sampled image frames. Suchextracted machine-readable images may include additional information(e.g., certain operational parameters) associated with the industrialdevices. In some cases, the extracted machine-readable images mayinclude additional data that may be used to improve data redundancy.

With the extracted machine-readable images from received streaming data,at block 458, the computing system may convert the machine-readableimages to determine data values. For example, the machine-readableimages may include QR code consists of black squares arranged in asquare grid on a white background, which may be read by an imagingdevice (e.g., a camera) coupled to the computing system. In someembodiments, the imaging device may use a QR reader tool to read the QRcode. The computing system may perform certain data processingoperations, such as using Reed—Solomon error correction to interpret themachine-readable images to convert the machine-readable images todetermine data values corresponding to live operational parametersassociate with the industrial devices. For example, the computing systemmay convert the machine-readable images and determine the data valuesbased on certain patterns presented in horizontal and verticalcomponents of the QR code.

At block 460, the computing system may store the data values in adatabase. For example, the computing system may store the data values astime series data corresponding to certain portions of image and/or audiodata representing changes associated with live operation parameters(e.g., temperature, pressure, motor speed) of an industrial device. Thecomputing system may use, for example, machine learning, to identify atrend associated with the operation of the industrial device and predictone or more future operational parameters of the industrial device basedon the identified trend. In some embodiments, the computing device mayidentify a missing image frame and retrieve the missing machine-readableimages associated with the missing image frames using a buffer of imagedatasets (e.g., a gather of high-density images consisting ofmachine-readable images) interlaced within the streaming image data.

In some embodiments, the interlaced image data may be used to verify thedata acquired via OCR or some other technology. In this way, the OCRtechnology may be tuned to allow for improved data acquisitionoperations.

In some embodiment, the different methods described herein may becombined to facilitate remote accesses to live data updates related tovarious equipment or devices of a system by multiple users across anorganization. For example, a control system may receive streaming datafrom industrial devices of an industrial system and query the industrialdevices for datasets associated with the streaming data. Based on thedatasets, the control system may generate machine-readable images andembed the machine-readable images within the streaming data. A datastreaming device may receive the streaming data embed with themachine-readable images. The data streaming device may determine viewaccess rights associated with a user of the multiple users based onauthenticated credentials of the user and route live streaming imagedata containing selected visualizations associated with selectedindustrial devices to a computing system associated with the user. Thelive streaming image data may include serial image frames and the usercredentials. Each image frame of certain portion(s) of the image framesmay include a subset of the machine-readable images corresponding to abuffer of the datasets.

The computing system may identify one or more data fields in the imageframes and extract one or more portions of the image frames based on thedata fields over a time period. From the extracted portions of the imageframes, the computing system may acquire one or more data field valuesand store the acquired data filed values in a time series database. Thecomputing system may access the time series database to receive timeseries data and identify one or more trends associated with one or moreindustrial devices. Based on the trends, the computing system maypredict one or more operational parameters of the one or more industrialdevices and send commands to the one or more industrial devices toadjust operations if the predicted operational parameters are outsideexpected ranges. In some embodiments, the computing system may extractthe machine-readable images from the live streaming image data andconvert the machine-readable images to determine data valuescorresponding to live operational parameters associate with theindustrial devices.

The techniques presented and claimed herein are referenced and appliedto material objects and concrete examples of a practical nature thatdemonstrably improve the present technical field and, as such, are notabstract, intangible or purely theoretical. Further, if any claimsappended to the end of this specification contain one or more elementsdesignated as “means for [perform]ing [a function]. . . ” or “step for[perform]ing [a function]. . . ”, it is intended that such elements areto be interpreted under 35 U.S.C. 112(f). However, for any claimscontaining elements designated in any other manner, it is intended thatsuch elements are not to be interpreted under 35 U.S.C. 112(f).

While only certain features of the embodiments described herein havebeen illustrated and described, many modifications and changes willoccur to those skilled in the art. It is, therefore, to be understoodthat the appended claims are intended to cover all such modificationsand changes as fall within the true spirit of the embodiments describedherein.

1. A system, comprising: a control system configured to control one ormore operations of one or more industrial devices in an industrialsystem, wherein the control system is configured to: receive streamingdata comprising one or more visualizations representative of one or morelive operational parameters associated with one or more industrialdevices, wherein the streaming data comprises a plurality of imageframes; identify a plurality of datasets associated with the streamingdata; generate a plurality of machine-readable images based on theplurality of datasets; embed the plurality of machine-readable imageswithin the plurality of image frames of the streaming data to generateupdated streaming data; and send the updated streaming data to acomputing system configured to extract the plurality of machine-readableimages from the updated streaming data.
 2. The system of claim 1,wherein the streaming data comprises metadata associated with theplurality of datasets.
 3. The system of claim 1, wherein the pluralityof datasets comprises data acquired by one or more sensors or producedby one or more other industrial devices in the industrial system.
 4. Thesystem of claim 1, wherein the control system is configured to query theone or more industrial devices for the plurality of datasets in responseto detecting one or more warnings, one or more alerts, or both.
 5. Thesystem of claim 4, wherein the one or more warnings, the one or morealerts, or both are triggered by detecting the one or more liveoperational parameters associated with the one or more industrialdevices are different from one or more device settings, above or belowone or more pre-determined threshold values, outside one or morethreshold ranges, or any combination thereof.
 6. The system of claim 1,wherein the plurality of machine-readable images comprises at least onebar code, at least one quick response code, or both.
 7. The system ofclaim 1, wherein the plurality of machine-readable images is configuredto convey information associated with one or more data valuescorresponding to the one or more live operational parameters associatedwith the one or more industrial devices.
 8. The system of claim 1,wherein the plurality of image frames is presented to the computingsystem at a frequency higher than 15 Hz.
 9. The system of claim 1,wherein the control system is configured to embed the plurality ofmachine-readable images within the plurality of image frames of thestreaming data based on a pre-determined order.
 10. The system of claim9, wherein the computing system is configured to extract the pluralityof machine-readable images from the updated streaming data based on thepre-determined order.
 11. The system of claim 1, wherein the computingsystem is configured to: determine one or more data values correspondingto the one or more live operational parameters associated with the oneor more industrial devices based on the plurality of machine-readableimages extracted from the streaming data; and store the one or more datavalues in a database.
 12. The system of claim 11, wherein the computingsystem comprises a quick code reader tool configured to determine theone or more data values, wherein the plurality of machine-readableimages comprises at least one quick response (QR) code.
 13. A method,comprising: receiving, via a control system, streaming data comprisingone or more visualizations representative of one or more liveoperational parameters associated with one or more industrial devices inan industrial system, wherein the streaming data comprises a pluralityof image frames; identifying, via the control system, a plurality ofdatasets associated with the streaming data; generating, via the controlsystem, a plurality of machine-readable images based on the plurality ofdatasets; embedding, via the control system, the plurality ofmachine-readable images within the plurality of image frames of thestreaming data to generate updated streaming data; and sending theupdated streaming data to a computing system.
 14. The method of claim13, wherein the computing system is configured to extract the pluralityof machine-readable images from the updated streaming data.
 15. Themethod of claim 13, wherein each image frame of a portion of theplurality of image frames comprises a subset of the plurality ofmachine-readable images, and wherein the subset of the plurality ofmachine-readable images corresponds to a buffer of a plurality ofdatasets.
 16. The method of claim 13, wherein a first image frame and asecond image frame of a portion of the plurality of image framescomprises a first subset and a second subset of the plurality ofmachine-readable images, respectively, and wherein the first subset andthe second subset of the plurality of machine-readable images compriseat least one matching machine-readable image.
 17. Non-transitory,computer-readable medium storing instructions that, when executed by oneor more processors, are configured to cause the one or more processorsto perform operations comprising: receiving streaming data via anetwork, wherein the streaming data comprises a plurality of imageframes; extracting a plurality of machine-readable images from theplurality of image frames, wherein the plurality of machine-readableimages corresponds to a portion of the plurality of image frames;determining one or more data values corresponding to one or more liveoperational parameters associated with one or more industrial devicesbased on the plurality of machine-readable images extracted from thestreaming data; and storing the one or more data values in a database.18. The non-transitory, computer-readable medium of claim 17, whereineach image frame of the portion of the plurality of image framescomprises a subset of the plurality of machine-readable images, andwherein the subset of the plurality of machine-readable imagescorresponds to a buffer of the one or more data values.
 19. Thenon-transitory, computer-readable medium of claim 17, wherein a firstimage frame and a second image frame of the portion of the plurality ofimage frames comprises a first subset and a second subset of theplurality of machine-readable images, respectively, and wherein thefirst subset and the second subset of the plurality of machine-readableimages comprise at least one matching machine-readable image.
 20. Thenon-transitory, computer-readable medium of claim 19, wherein the firstimage frame and the second image frame corresponds to a pre-determinedorder with respect to the plurality of image frames.